feat: add --output-format json to test command - #772

Background

Adds structured JSON output to the test command for CI/CD integration and programmatic parsing of test results.

This addresses a common need for organizations integrating Panther rule testing into automated workflows — the existing text output is great for humans but fragile to parse programmatically. With --output-format json, a single JSON object is printed to stdout containing summary counts, per-detection test results, failures, invalid specs, and skipped tests. All logging is redirected to stderr so stdout remains clean, parseable JSON.

Closes #634

Usage Examples

panther_analysis_tool test \
  --path rules/ \
  --filter RuleID=Crowdstrike.Detection.Passthrough \
  --output-format json 2>/dev/null

{
  "summary": {
    "path": "rules/",
    "total": 1,
    "passed": 1,
    "failed": 0,
    "invalid": 0,
    "skipped": 0
  },
  "results": {
    "Crowdstrike.Detection.Passthrough": [
      {
        "name": "Low Severity Finding",
        "passed": true,
        "errored": false,
        "functions": [
          {"name": "rule", "status": "pass", "output": "true"},
          {"name": "title", "status": "pass", "output": "Crowdstrike Alert: NGAV on macbook"},
          {"name": "severity", "status": "pass", "output": "LOW"},
          {"name": "alertContext", "status": "pass", "output": "{\"cid\": \"11111111...\", \"Technique\": \"PUP\"}"}
        ]
      },
      {
        "name": "High Severity Finding",
        "passed": true,
        "errored": false,
        "functions": [
          {"name": "rule", "status": "pass", "output": "true"},
          {"name": "title", "status": "pass", "output": "Crowdstrike Alert: Ransomware on workstation"},
          {"name": "severity", "status": "pass", "output": "CRITICAL"}
        ]
      }
    ]
  },
  "failed": {},
  "invalid": [],
  "skipped": []
}

panther_analysis_tool test \
  --path rules/ \
  --filter RuleID=Crowdstrike.Detection.Passthrough \
  --output-format json 2>/dev/null | jq '.summary'

{
  "path": "rules/",
  "total": 1,
  "passed": 1,
  "failed": 0,
  "invalid": 0,
  "skipped": 0
}

panther_analysis_tool test \
  --path rules/ \
  --filter RuleID=Crowdstrike.Detection.Passthrough \
  --output-format json 2>/dev/null \
  | jq '.results["Crowdstrike.Detection.Passthrough"][] | select(.name == "Low Severity Finding") | {name, passed, title: (.functions[] | select(.name == "title") | .output), alertContext: (.functions[] | select(.name == "alertContext") | .output)}'

{
  "name": "Low Severity Finding",
  "passed": true,
  "title": "Crowdstrike Alert: NGAV on macbook",
  "alertContext": "{\"cid\": \"11111111...\", \"Technique\": \"PUP\"}"
}

Changes

New CLI option:

• --output-format {text,json} on the test command (default: text), backed by an OutputFormat(str, Enum) so Typer provides validation and shell completion out of the box
• Configurable via PANTHER_OUTPUT_FORMAT environment variable

JSON output functions:

• _print_json_output() — serializes full test results (summary, per-detection results, failures, invalid specs, skipped tests) as compact JSON to stdout
• _print_json_error() — emits a valid JSON error envelope on early-exit paths (empty specs, no filter match) so CI consumers always get parseable output
• _serialize_test_result() / _serialize_function_result() — convert internal dataclass results to JSON-safe dicts

Robustness fixes identified during code audit:

• Clamp num_passed to max(0, ...) to prevent negative counts when invalid_specs includes non-detection items
• Replace print() with logging.error() in setup_data_models to avoid stdout pollution in JSON mode
• Guard handler.setStream() with isinstance(handler, logging.StreamHandler) to avoid AttributeError on non-stream handlers (e.g., NullHandler)
• Defensive error access in _serialize_function_result handles both dict and string error values
• _buffer_error_result() ensures tests that throw exceptions still appear in JSON results, keeping failed and results keys consistent
• Suppress print() calls in setup_run_tests and run_tests when buffering results for JSON output

Version bump: 1.5.2 → 1.6.0

Testing

Authored by Cursor Agent using claude-4.6-opus-max